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Overview 


Achieving the Dream (ATD) is a national nonprofit organization dedicated to 
improving outcomes among community college students, especially low-income students 
and students of color. A central ATD strategy is to promote a “culture of evidence,” in 
which colleges collect, analyze, and make decisions based on information about students in 
order to inform improvements in practice. 

This report examines the use of data on students by faculty, administrators, and 
student services staff at six Washington State colleges that joined ATD in 2006-2007. 
Surveys were administered to faculty and administrators in 2007 and to faculty, 
administrators, and student services staff in 2010. We analyzed the survey data in order to 
understand differences in data use between the three groups (faculty, administrators, and 
student services staff) and to understand whether there were changes in the frequency and 
extent of data use between the two survey waves at the participating colleges. The 
following are the main findings of our analysis: 

• Administrators were more frequent and intensive users of 
student outcomes data and research from their college than 
faculty or student services staff. 

• Most faculty and student services staff did not examine student 
progression or use outcomes data on a regular basis. 

• Although the frequency with which faculty used data on 
student progression and completion did not increase between the 
two waves of the survey, their use of data to inform teaching- 
related decisions did increase. 

• Most faculty did not use administrative data, such as data from 
their college’s student information system. 

• Faculty members’ use of data was correlated with their 
department’s use of data. 

Overall, there was broad use of data by administrators and of certain types of data 
by faculty and student services staff, but there remain opportunities for increasing the use 
of student data in the Washington State ATD colleges. To further promote the use of data 
in support of improved student success, ATD colleges in Washington State and elsewhere 
should consider ways to better connect the data collected and reported to the primary focus 
of faculty on instruction, and they should consider ways to engage student services staff 
more in the use of data on student progression, given their interest in student retention. 
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1. Introduction 


In 2004, Lumina Foundation for Education and seven partner organizations 
created the Achieving the Dream (ATD) reform initiative to improve outcomes among 
community college students, especially low-income students and students of color. 
Twenty-six colleges in five states (referred to as the “Round 1” colleges) joined ATD in 
2004-2005. Today, ATD is a national nonprofit dedicated to increasing the number of 
students who earn postsecondary credentials with marketplace value, and more than 150 
colleges in 30 states and the District of Columbia participate in the initiative. 

The Round 1 colleges as well as six Round 3 Washington State community and 
technical colleges (which joined ATD in 2006-2007) have been part of an ongoing 
evaluation of the ATD initiative, which is based on a set of principles designed to change 
how community colleges function. The ATD model for institutional change includes four 
principles: (1) committed leadership, (2) use of evidence to improve programs and 
services, (3) broad engagement of stakeholders, and (4) systemic institutional 
improvement. With the second principle, ATD calls for the development of a “culture of 
evidence” in which colleges collect, analyze, and make decisions based on information 
about students in order to inform improvements in practice. In this report, we examine 
perceptions and behaviors related to the development of a culture of evidence among 
personnel at the six Round 3 Washington ATD colleges. Our analysis is based on surveys 
we conducted in 2007-2008 and in 2010-201 1 on the use of data by college personnel at 
these Washington ATD colleges. 

One premise of ATD is that using data to understand patterns of student 
progression and completion will help community college faculty and staff to identify 
barriers to student success and develop strategies to overcome those barriers. ATD 
encourages faculty and staff at participating colleges to examine the progress and 
outcomes of their students and gaps in achievement by race/ethnicity and income. With 
our study, we were interested in assessing the use of the sorts of data that ATD promotes: 
completion of developmental education courses, persistence from term to term and year 
to year, the use of learning measures other than grades, and student outcomes such as 
graduation and transfer to a four-year institution. 
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Our reasons for examining ATD-related data were twofold. First, a college’s 
participation in ATD should result in changes in behavior and attitudes relating to data. 
Second, in the years since ATD began, the national conversation about community 
colleges has shifted from student access to student persistence and graduation. To 
increase rates of persistence and graduation, it is essential that faculty, student services 
staff, and administrators at community colleges are actively engaged in understanding the 
outcomes of the students at their college. 

The current report extends discussion found in an earlier report on the first wave 
of the survey (Jenkins & Kerrigan, 2008).^ It also complements another report that 
examined the implementation of ATD in the Washington State colleges more broadly, 
summarizing the findings of a cross-sectional analysis that compared data use by faculty, 
administrators, and student services staff and discussing noteworthy changes in data use 
by faculty and administrators (Jenkins, Wachen, Kerrigan, & Mayer, 2012). The main 
findings of the current report are as follows: 

• Administrators were more frequent and intensive users of 
student outcomes data and research from their college than 
faculty or student services staff. 

• Most faculty and student services staff did not examine student 
progression or use outcomes data on a regular basis. 

• Although the frequency with which faculty used ATD-relevant 
data did not increase between the two waves of the survey, their 
use of data to inform teaching-related decisions did increase. 

• Most faculty did not use administrative data, such as data from 
their college’s student information system. 

• Faculty members’ use of data was correlated with their 
department’s use of data. 

In the next section, we describe the study’s methodology. Then, we discuss the 
patterns of data use, including the frequency with which faculty, administrators, and 
student services staff used specific kinds of data and the intensity with which they used 
data to inform decisions and plan. We also explore how frequently faculty, 
administrators, and student services staff engaged in organized discussions about 


* The first wave of the survey was administered at 28 Round 1 colleges and seven additional Round 3 
colleges, in addition to the six Round 3 Washington colleges discussed in this report. 
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achievement gaps. Although we cannot establish casual relationships in this analysis, we 
do explore possible explanations for the patterns of data use observed in the findings. In 
the conclusion, we offer recommendations for colleges seeking to use data to improve 
student outcomes. 


2. Method 


2.1 Study and Instrument Design 

In 2007, researchers at CCRC and MDRC developed survey instruments to 
capture faculty and administrators’ perceptions and behaviors relating to the use of data 
at ATD colleges. The instruments were developed in consultation with individuals 
involved in ATD, including coaches and data facilitators working with the colleges, and 
with other community college experts. The surveys were designed to permit cross- 
sectional analyses and to capture changes that occurred between the first and second 
waves of the survey. The original two instruments — a faculty survey and an administrator 
survey — were pilot tested at three ATD colleges that did not participate in the study. 

In 2010, we revised the faculty and administrator instruments. Some questions 
were refined to capture more detail or deleted because they provided data that were not 
particularly interesting or useful. To capture more detail, we changed some yes/no 
questions to Likert items that captured frequency or extent on a 7-point scale. We also 
added a few questions on program assessment practices and about the involvement of 
part-time faculty that were not part of the 2007-2008 survey, although we sought to keep 
the questions as consistent with the first wave of the survey as possible to facilitate 
analysis of changes over time. In addition, as the data from the first survey wave and 
from other research on Round I colleges emerged, we recognized the key role of student 
services staff in issues of student persistence. As a result, we added a student services 
instrument in the second wave that was modeled on the administrator instrument. 

Each instrument contained approximately 100 items, most of which overlapped, 
relating to three topics: data use, data accessibility, and familiarity with and involvement in 
ATD. Most items used a 7-point Likert scale; some questions used a yes/no format. We 
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asked about the frequency with which the respondents used 20 types of data, including 
admissions data; graduation, transfer, and retention rates; grades; and research-related data, 
such as Community College Survey of Student Engagement findings and college reports. 
We also asked about the frequency with which respondents used data disaggregated by 
race/ethnicity and income. In addition to frequency of data use, we were interested in the 
intensity of data use for the decision making and planning. We included questions about the 
accessibility of data to understand the sources of data and perceptions about barriers to data 
use. Finally, we collected information about participants’ engagement with ATD because 
we hypothesized that those who were most active with the initiative would also be the most 
active users of data. The instruments included a set of demographic questions about the 
respondent and his or her role at the college. 

The Human Resources Research Organization developed a web interface for the 
survey and administered both waves of the survey. MDRC and CCRC worked closely 
with the Human Resources Research Organization during the administration of the 
survey and were responsible for coordinating the survey process with participating 
colleges and addressing questions about content. 

2.2 Sample and Survey Response Rates 

To capture individuals in decision-making positions, the sampling frame included 
all full-time administrators at the director level and above, full-time faculty, and full-time 
student services staff. We defined these categories and worked with the institutional 
research offices at each college to gather the contact information for these individuals. 

The final sample size in the 2010-201 1 wave of the survey was 679, including 
424 faculty, 131 administrators, and 124 student services staff. The overall response rate 
was 69 percent; response rates for each college are provided in Table 1. 

The respondents in wave 2 included faculty and administrators — some of whom 
had participated in wave 1 and some of whom had not (see Table 2) — and student 
services staff, none of whom were included in wave 1. Although only 168 faculty and 64 
administrators participated in both waves of the survey, we discuss changes over time in 
this report, when statistically significant, because they represent differences in the 
behavior and perceptions of these functional groups in the Washington community and 
technical colleges. 
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Table 1 

Survey Response Rates by College 


College 

2007- 

-2008 


2010-2011 


Faculty 

(%) 

Administrators 

(%) 

Faculty 

(%) 

Student Services 
(%) 

Administrators 

(%) 

WA-A 

45 

61 

69 

69 

94 

WA-B 

64 

93 

67 

66 

79 

WA-C 

66 

77 

64 

65 

77 

WA-D 

74 

80 

79 

100 

72 

WA-E 

74 

93 

62 

100 

85 

WA-F 

46 

81 

49 

76 

91 

Mean 

62 

81 

65 

71 

83 


Table 2 

Participants in Each Wave of the Survey 


Wave 

Faculty 

Administrators 

Wave 1 only 

222 

117 

Wave 2 only 

257 

69 

Both waves 

168 

64 


2.3 Data Analysis 

We analyzed the survey data in order to understand differences in data use 
between the three groups (faculty, administrators, and student services staff) and to 
understand whether there were changes in the frequency and extent of data use between 
the two survey waves at the participating colleges. To capture differences between the 
three groups in the second wave of the survey, we used the Wilcoxon-Mann- Whitney test 
to determine the significance of all ordinal variables and the chi-square test for all binary 
variables. To capture differences between the two waves, by group, we used the chi- 
square test and the t-test. We employed Fisher’s test as the significance test for equality 
of each pair of correlations between the two waves of the survey. 

Our analysis focused on patterns of data use by faculty, administrators, and 
student services staff and what might influence that use across the Washington State 
ATD colleges in the second wave of the survey. Where applicable, we also addressed 
statistically significant changes in patterns of data use by faculty and administrators 
between the survey waves. We do not discuss changes in data use by student services 
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staff because they were not included in the first wave of the survey. For more information 
about differences in data use among the colleges, see Jenkins et al. (2012, chapter 4). 


3. Patterns of Data Use 

In this section, we present notable findings from the responses of faculty, 
administrators, and student services staff at the Washington ATD colleges to survey 
questions about the use of data and research on students by themselves, their 
departments, and their colleges. We focus particularly on frequency and intensity of data 
use. Survey questions related to the former used a Likert scale that ranged from “never” 
to “more than 6 times per year”; survey questions related to the latter used a Likert scale 
that ranged from “not at all” to “a lot.” We designated those who used specific kinds of 
data at least four times per year as frequent users and those who reported using data “a 
lot” as intensive users. 

3.1 Frequency of Data Use 

Survey participants were asked about their use of data relating to conventional 
measures of enrollment and grades as well as types of data that ATD prioritizes, such as 
retention rates, graduation rates, transfer rates, and developmental education course 
completion rates. Participants used some types of data more than others, and frequency of 
data use varied by respondents’ roles at the college. The following are notable findings 
related to frequency of data use: 

• Faculty, like administrators and student services staff, were 
very focused on enrollment data. Almost 60 percent looked at 
enrollment data frequently, and less than 10 percent reported 
reviewing it less than once a year. 

• More than two thirds of faculty used data on grades at least 
four times a year. 

• Only 30 percent of faculty reported using data on measures of 
learning other than grades (such as student learning outcomes, 
licensure exams, achievement tests, or other competency 
assessments) at least four times a year. More than half reported 
using such data less than once a year (see Figure 1). 
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• Most faculty were infrequent users of data on student retention 
and graduation rates. Less than 20 percent used such 
information intensively, and between 20 and 30 percent used it 
less than once a year. 

• Although a majority of faculty used research by the college 
sometimes, only 10 percent were intensive users, and 20 
percent never used it. 

• Faculty were more inclined to use outside research; 34 percent 
were frequent users, and only 12 percent reported using such 
information less than once a year. 

• Student services staff were less likely than administrators to 
use data on retention and graduation rates, although 23 to 30 
percent used such information frequently (see Figure 1). 

• One third of student services staff did not use research conducted 
by the college, and nearly half did not use outside research. 

• Over 80 percent of administrators were frequent users of data 
on enrollment and budgets. 

• Less than 40 percent of administrators regularly used data on 
student retention rates and graduation rates (see Figure 1). 

• Administrators were much more likely than faculty and student 
services staff to be frequent users of research produced by the 
college. Half of the administrators who responded to the survey 
were frequent consumers of such information, and nearly half 
were frequent users of outside research (see Figure 2). 

In general, the types of data for which we saw increased frequency of use by 
faculty — placement test scores, grades, budgets — were not necessarily those emphasized 
by ATD. Use of data on retention rates, graduation rates, transfer rates, and developmental 
course completion rates did not change markedly. Given ATD’s emphasis on implementing 
strategies that require the active involvement of faculty and student services staff, we were 
surprised to find that most faculty and student services staff were not looking at data on 
student progress and success. In the case of graduation rates, however, one could argue that 
they do not change frequently enough to warrant frequent use. 
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Figure 1 

Frequent Users of ATD-Related Data 
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Note. We define "frequent users" as respondents who indicated that they used a particular type of data at least four 
times per year. 



Figure 2 

Frequent Users of Research 
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Note. We define "frequent users" as respondents who indicated that they used a particular type of data at least four 
times per year. 
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3.2 Use of Data on Achievement Gaps 

Given ATD’s focus on developing a culture of evidence and helping low-income 
students and student of color, we asked survey participants about the frequency with 
which they used data on gaps in student achievement and the frequency with which they 
used data broken down by students’ race/ethnicity or income level. The following are our 
main findings: 


• Although more than half of faculty members surveyed used 
data on achievement gaps at least once a year, less than 15 
percent used that information frequently, and about a quarter of 
respondents used it less than once a year (see Figure 3). 

• Administrators were much more likely than faculty to be 
frequent users of data on gaps in students’ academic 
achievement and achievement data broken down by race and 
income. As shown in Figure 4, there was a statistically 
significant increase in administrators’ use of data on 
achievement gaps between the waves of the survey. 

• Although student services staff were slightly more likely than 
faculty members to be frequent users of data on gaps in 
students’ academic achievement, over half of the student 
services staff who participated in the survey reported that they 
did not use data on gaps in students’ academic achievement. A 
higher proportion of student services staff reported frequent use 
of data disaggregated by race and income (see Figure 3). 

In terms of changes between the two waves of the survey, ATD’s emphasis on 
outcomes and disaggregated data does not seem to have significantly influenced the 
frequency of use of these types of data among faculty. The increase in administrators’ use 
of data on academic achievement and student data disaggregated by race and income is 
promising, but further attention must be paid to how to reach faculty. 
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Figure 3 

Frequent Users of Disaggregated Student Data 
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Note. We define "frequent users" as respondents who indicated that they used a particular type of data at least four 
times per year. 


Figure 4 

Changes in Frequent Use of Data on Achievement Gaps by Faculty and Administrators 



Note. We define "frequent users" as respondents who indicated that they used a particular type of data at least four 
times per year. 
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3.3 Use of Data to Inform Decisions 

In addition to understanding the frequency with which respondents used particular 
types of data, we were also interested in understanding the intensity of data use for 
informing decisions about teaching and for other purposes by the respondents, their 
departments, and their colleges. 

Although faculty did not increase their use of data on student progression, 
completion, and achievement gaps between 2007-2008 and 2010-2011, they did increase 
the extent to which they used data to inform teaching-related decisions. There was an 
increase in the proportion of faculty who reported using data “a lot” to inform curricular 
decisions, teaching, and advising and to identify areas at the college needing 
improvement. In the first wave of the survey, about one in five faculty members reported 
using data and research on students “a lot” to inform their own decisions about 
curriculum, teaching, advising, and identifying struggling students. During the second 
wave, at least one in four faculty members reported doing so. For some items, a third of 
the faculty were identified as intensive users of data for teaching-related decisions (see 
Figure 5). 

The majority of faculty, administrators, and student services staff reported that 
their colleges used data for decision making. However, that finding is tempered by 
respondents’ indications that less than 50 percent believed their college used data “a lot” 
for planning. 

• Most faculty members reported that their departments used 
data to inform decisions about curriculum, instruction, 
planning, and program improvement, although only one quarter 
to one third indicated that their departments used data “a lot” 
for these purposes (see Figure 6). 

• Student services staff were somewhat more likely than faculty 
or administrators to indicate that their departments used data “a 
lot” for planning, budgeting, and identifying areas for 
improvement (see Figure 6). This is not consistent with other 
areas of the survey where we found that student services staff 
used data less than faculty and administrators. 

• Although administrators’ reports indicated that their 
departments did not use data for teaching-related purposes as 
intensively as faculty’s, they did use data for planning 
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purposes. As Figure 6 illustrates, around 30 percent of 
administrators reported that their departments were intensive 
users of data to inform evaluation, strategic and long-term 
planning, budgeting, and identification of areas at the college 
needing improvement. 

• More than 88 percent of faculty, administrators, and student 
services staff believed that their college used data at least to 
some extent in making decisions about program planning and 
evaluation, identifying areas for improvement, evaluating 
student services, long-term planning, and budgeting. However, 
far fewer respondents indicated intensive use of data by the 
college (see Figure 7). 

• Only about 40 percent of respondents believed their college 
used data a lot for long-term planning, and less than 40 percent 
believed that their college used data intensively for program 
planning and evaluation. 


Figure 5 

Intensive Use of Data for Teaching-Related Purposes Among Faculty 
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Figure 6 

Intensive Use of Data by Departments for Planning Purposes 
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Note. We define "intensive users" as respondents who indicated that they used the given type of information "a lot." 


Figure 7 

Intensive Use of Data by Colleges for Planning Purposes 
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Note. We define "intensive users" as respondents who indicated that they used the given type of information "a lot." 
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3.4 Frequency of Participation in Organized Discussions 

We were also interested in the frequency with which college personnel 
participated in organized discussions at the college about improving academic 
achievement or closing achievement gaps and about the academic needs and performance 
of students of color, low-income students, and academically underprepared students. 
Formal discussions about student achievement data should indicate awareness that gaps 
exist and are a precursor for taking appropriate action. 

We found that administrators were much more likely than faculty and student 
services staff to participate frequently in organized discussions on any of these topics (see 
Figure 8). Less than one third of the faculty and less than one quarter of the student 
services staff reported engaging in such organized discussions at least four times a year. 
Additional outreach may be needed — even in colleges that are engaged in an initiative to 
improve student success and close achievement gaps — to regularly involve personnel 
other than administrators in organized conversations about academic achievement. 


Figure 8 

Frequent Participation in Organized Discussions About Students by Topic 
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Note. We define "frequent users" as respondents who indicated that they used a particular type of data at least four 
times per year. 
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ATD seeks to promote success among low-income students and student of color, 
in part by enabling conversations about difficult topics such as differences in outcomes 
by race and income. Thus, we expected to see broad engagement in organized discussions 
about academic achievement and achievement gaps. We were encouraged by the notable 
increase between the two waves of the survey in the percentage of administrators 
participating frequently in organized discussions about achievement gaps. At the same 
time, despite three years of participation in ATD, faculty and student services staff were 
on average no more likely to participate frequently in such discussions in 2010-201 1 than 
they had been in 2007-2008. These findings are consistent with the other indications that 
faculty and student services staff in the Washington ATD colleges were not as engaged in 
use of data and research as were administrators. With few exceptions (i.e., the use of 
learning measures besides grades and the use of data for planning purposes), 
administrators consistently used data more frequently than faculty or student services staff. 
In the following section, we explore possible explanations for these patterns of data use. 


4. Influences on Data Use 

4.1 Facilitators and Inhibitors of Data Use 

A number of our survey questions addressed possible influences on data use. We 
were concerned with influences that might promote or inhibit the frequency and intensity 
of data use within a college, including departmental practices, sources of data on 
students, the accessibility of data, possible barriers to using data, and participation in 
training in the use of data. Our findings in this area may inform our interpretations of the 
data use patterns described in the previous section. 

Accessibility of data. One core aspect of data use is the availability of data. Thus, 
we asked respondents about the sources they used for information on groups of students. 
Overall, faculty, administrators, and student services staff reported feeling that data were 
accessible, but the groups differed in some of their perceptions in this area. 

• Relatively few faculty members — fewer than 1 in 5 — used their 
college’s student information systems. In comparison, over 40 
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percent of administrators and student services personnel used 
their college’s student information system (see Figure 9). 

• Despite the growth of web-enabled reports at the colleges 
during the period under study, there was no change in types of 
data used. Only 20 percent of faculty used data on their 
college’s website. 

• The main consumers of research from college institutional 
research (IR) offices were administrators. Just over a third of 
faculty and student services staff requested reports from the 
IR office, compared with over two thirds of administrators. 

The proportion of administrators making requests for 
information from their IR office increased between the two 
waves of the survey. 

• More than 70 percent of all respondents reported that the IR 
office was responsive to requests. Fewer respondents, however 
(approximately 50 percent) agreed that the IR office was 
adequately staffed (see Figure 10). 

• Half of faculty members agreed that their college’s data were 
available in a user-friendly format, and almost two thirds felt 
that the reports they received from the college were clear (see 
Figure 10). 

Our findings have implications for ATD and other reforms that emphasize 
administrative data. Between the first and the second waves of the survey, there was an 
increase in the perceptions of faculty and administrators of the accessibility of data. It is 
possible that this was related to ATD. However, it remains unclear why, if faculty felt 
that data were more accessible than in the past, there was no increase in the use of data 
by faculty. 
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Figure 9 

Data Sources Used by Survey Respondents 
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Figure 10 

Perceptions About Data Accessibility 
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Barriers to data use. Although most survey respondents found data to be 
accessible, some identified reasons for not using data. 

• Only 14 percent of faculty said data use was not their 
responsibility, 16 percent said they lacked the skills, and 17 
percent said they did not trust the data. 

• Over one third of faculty — more than any other group — 
reported that they were too busy to use the data (see Figure 11). 

• About one in five faculty reported that they found the data 
irrelevant to their role as teachers. 

• Just over a quarter of the administrators and student services 
staff responded that the available data on students were not 
relevant to their job functions (see Figure 11). 

• Student services staff were evenly divided in their perceptions 
of whether using data was part of their responsibilities: 40 
percent of the respondents agreed that it was not part of their 
job responsibilities, and 43 percent disagreed. 

It is interesting to note, considering that faculty did not use data on students as 
much as administrators or student services personnel, that faculty were by far the least 
likely of the three groups to say that using data was not their responsibility. 


Figure 11 

Reasons for Not Using Data 
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Training and involvement in ATD. Evidence from previous studies of 
community college institutional research led us to hypothesize that training on how to use 
data is an important influence on the use of data (see Merest & Jenkins, 2007). Thus, we 
asked survey participants about the frequency of their participation in training on either 
institutional research or assessment and evaluation. Participation in training varied among 
faculty, student services staff, and administrators. 

• More than half of the administrators had participated in 
training relating to institutional research or assessment and 
evaluation in the past year, and less than a quarter reported 
never having participated in such training. 

• Almost half of the student services staff had never participated 
in such training. 

• More than half of faculty reported having participated in 
training relating to assessment and evaluation in the past year, 
and fewer than one in five reported never having participated in 
training on assessment and evaluation. About 40 percent of the 
faculty had participated in training on institutional research, 
and about a third had never participated in such training. 

• Of those who participated in the survey, about 20 percent of the 
faculty and student services staff were involved in an ATD 
team or in implementing an ATD strategy. Involvement among 
administrators was higher: 47 percent of the administrators 
who responded reported participating in an ATD team, and 37 
percent were involved in implementing an ATD strategy. 

Summary and discussion. Our findings regarding the perceived barriers to data 
use provide only limited insight into the patterns of data use we observed. Most faculty, 
administrators, and student services staff indicated that the data provided by their college 
were accessible and trustworthy. There was some evidence that data provided by the 
college had become more accessible between the first and second surveys. The vast 
majority of faculty (86 percent) indicated that using data on students was part of their 
responsibility, and only a small percentage indicated that they did not have the necessary 
skills or did not trust the data. Moreover, over half of faculty indicated that they had 
participated in training on assessment and evaluation in the past year, and a substantial 
number had also recently completed training related to institutional research. And yet, we 
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observed that most faculty did not frequently use data on students. Over a third of faculty 
indicated that they were too busy to use data on students. In our fieldwork, we heard from 
both faculty and administrators that the fiscal pressures facing community colleges in 
Washington State and elsewhere were leading many faculty members to feel overworked 
(Jenkins et ah, 2012). So, a lack of time could explain, at least in part, why we saw a 
relatively low overall level of data use among faculty. 

Among student services staff, forty percent of those surveyed indicated that they 
did not perceive using data on student outcomes as their responsibility. Moreover, nearly 
half had never participated in training related to institutional research or assessment and 
evaluation. These findings are more consistent with the patterns of low data use we 
observed among student services personnel. 

In the next section, we examine more closely the association between barriers to 
using data as well as other factors and the extent of data use by faculty, administrators, 
and student services staff in the Washington ATD colleges. 

4.2 Associations Between Indicators of Data Use and Other Factors 

To further explore influences on data use, we created indicators of data use from 
multiple survey questions at three levels: college, department, and individual respondent. 
These indicators enabled us to examine the relationship between data use and factors 
such as college and department practices that we hypothesized were associated with data 
use at participating colleges. 

We sought to understand how these three levels may relate and influence each 
other. We examined both the correlations between indicators and changes in the 
correlations between indicators over time. Although we ran many correlations based on 
our hypotheses, we only discuss those that are statistically significant. 

Associations with colleges’ use of data. We found the fewest correlations at the 
college level. 

• Colleges’ use of data was weakly related to other indicators or 
practices at the department or individual level. 

• Survey respondents’ perceptions of their leaderships’ 
commitment to using data were related to the extent to which 
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data on students were perceived to influence policy decisions 
made by college- wide committees. 

• Colleges’ practices regarding data use and evaluation were 
more strongly associated with frequent use of data among 
student services staff than with frequent use of data among 
faculty and administrators. 

Associations with departmental use of data. We found several associations 
between departmental practices and data use among faculty and administrators. 

• Faculty were more likely to use data and research in teaching- 
related decisions if they were in departments where decisions 
were made using data. 

• Between the two waves of the survey, we saw an increased 
association between various indicators of data used by faculty, 
such as the frequency of use of student outcomes data or 
disaggregated data, and the extent to which their departments 
used data to evaluate the effectiveness of their programs and 
services. These relationships were generally small but negative 
in the first wave, but in the second wave, they were positive. 

• More so than faculty, administrators in departments that were 
perceived to use data in decisions were more likely to use data 
on student outcomes. 

• Administrators in departments that were perceived to use data 
in decisions were also more likely to participate in organized 
discussions on achievement gaps and the needs of 
disadvantaged students. 

Associations with respondents’ use of data. There were also a number of 
correlations at the level of the individual respondent. Of particular interest were the 
associations between participation in organized discussions about students and data use, 
which we observed across all three groups, and which increased between the waves of the 
survey. These findings are promising for ATD and other efforts to promote the use of 
evidence in decision making at community colleges. 

• Faculty who participated in organized discussions about the 
needs of students of color and low-income students were much 
more likely to use data on student outcomes more frequently. 
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• Faculty who participated in discussions about closing 
achievement gaps used data disaggregated by student 
characteristics more frequently. 

• Administrators who participated in organized discussions about 
closing achievement gaps or about the needs of disadvantaged 
students were more likely to use student outcomes data. 

• Student services staff who participated in organized 
discussions about achievement gaps and disadvantaged 
students were more likely to use student outcomes data and to 
use data in decisions related to their work. 

• Participation in training on institutional research or assessment 
and evaluation was correlated with frequency of data use for 
faculty, administrators, and student services staff. 

• Respondents who participated in ATD activities were more 
likely than those who did not participate to use data frequently. 

• Respondents who participated in ATD activities were more 
likely than those who did not participate to take part in 
organized discussions about student achievement gaps and 
about disadvantaged students. 

Summary and discussion. Like the first round of the survey, the second round 
revealed a relationship between college practices and departmental and respondent 
practices, but the relationship remained weak. There was a stronger relationship between 
departments’ use of data and faculty members’ use of data and research in teaching- 
related decisions. Although college practices did not seem to significantly influence 
faculty and administrators, departmental practices did appear to influence faculty. This 
suggests that initiatives such as ATD may promote evidence-based decision making more 
effectively by focusing their efforts on deans and department heads. 


5. Continuing to Build a Culture of Evidence in ATD Colleges 

ATD promotes the use of evidence on student progression and outcomes to 
inform improvements in programs and services at community colleges. At the six 
Washington State colleges that joined ATD in 2006-2007, after five years of 
participation in the initiative, there were some increases among faculty in the use of data 
for teaching-related decisions and for decision making. We also saw an increase in the 
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use of student achievement data and data disaggregated by race and income among 
administrators. However, there was not an increase in faculty members’ use of data on 
student progression and completion, and in general, only a minority of faculty and 
student services staff used such data intensively. Although faculty did report that access 
to information on students had improved, only 16 percent of faculty accessed data on 
students from their college’s student information system, and faculty were more likely to 
rely on research on students from outside the college rather than analyses by institutional 
research staff. 

Our findings suggest that the most frequent and intensive users of student 
progression and outcomes data and data disaggregated by student characteristics were not 
those closest to students. Administrators continued to use all data except grades and other 
learning measures more frequently than did faculty and student services staff. The survey 
results suggest that faculty were more focused on what was happening within their own 
classrooms than on students’ progress beyond the courses they taught — and accordingly, 
we did find that the use of data to inform teaching increased between the two waves of 
the survey. This somewhat narrow focus does not necessarily imply that faculty were not 
interested in student success. Rather, in light of prior research, it suggests that there 
should be a greater focus on instruction and learning outcomes in the data that colleges 
collect and report (Jenkins et ah, 2012). 

Although faculty were more likely to report that they were “too busy” to use data 
on students, they were the least likely of the groups surveyed to agree that using data on 
students was not their job responsibility. One reason why the data on student progression 
may not have been of much interest to faculty was that the data were generally presented 
as institutional aggregates. To the extent that colleges can present data on students by 
program, faculty find the data more relevant and interesting (Jenkins & Cho, 2012). The 
finding that data use by individual faculty was correlated with data use in their 
departments reinforces the idea that, whenever possible, data should be disaggregated by 
program and department. 

Few faculty members used their college’s student information system, college 
website, departmental databases, or state databases. Administrators were by far the most 
frequent users of college reports and IR office information, and we saw a notable increase 
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in the proportion of administrators making requests for information from their IR offices 
between the two survey waves. This is further indication that the use of data and research 
increased among administrators but not among faculty in general, even though faculty 
reported that they felt data were more accessible than in the past. The fact that faculty 
members reported limited use of administrative data has implications for ATD and other 
reforms emphasizing such data. It is important that reform leaders consider how to 
connect what goes on in the classroom with broader student outcomes. 

Given the relationship between departmental use of data and faculty use of data, 
an opportunity exists for departments to help faculty focus on patterns of student 
progress. This approach makes sense because student progress is not just a college issue 
but also a departmental issue; students must move through multiple levels of math, 
English, and program- specific course sequences in order to complete a degree or 
certificate. In our fieldwork for a complementary study on ATD implementation at these 
colleges (see Jenkins et ah, 2012), we saw examples of departments using data on student 
progression to help set policies about what courses should be prerequisites for others. 

Further examination is needed of data use by student services staff, who are 
among the personnel at a community college who work most closely with new students 
and who are actively involved in efforts to retain students. The survey indicates that they 
used data less frequently than administrators but more frequently than faculty. They were 
among the heaviest users of their college’s student information system and departmental 
databases, but they were not frequent users of research — 40 percent of student services 
staff did not use research conducted by the college, and nearly half did not use outside 
research. Interestingly, they were the most likely of the groups surveyed to report that 
using data on students was not part of their job responsibilities. Over half of the student 
services staff members who participated in the survey reported not using data on gaps in 
students’ academic achievement, although they were slightly more likely than faculty to 
be intensive users of such data. In general, like faculty, many student services staff 
members were not looking at the “big picture” on student progress and success. Because 
40 percent of student services staff indicated that it was not their responsibility to review 
data on students, college leaders should consider the role student services staff play in 
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their improvement initiatives and how they can convey the message that reviewing data 
on students on a regular basis is a core responsibility. 

We observed a number of positive changes between the two waves of the survey 
and see opportunities to learn from those results. First, the across-the-board increases in 
the use of data on budgets suggest that behaviors within colleges are influenced by 
contextual factors and by the college leadership. Between the first and second survey 
waves, there was a decline in state funding of community colleges in Washington State, 
and colleges were forced to review their budgets for areas to cut or tighten. This 
phenomenon was not limited to administrators; faculty also increased the frequency with 
which they reviewed data on budgets and the extent to which student data informed 
decisions about budgeting. Although budget data is not an area that ATD prioritizes, this 
finding is nonetheless encouraging because it shows that with the right incentives, college 
personnel — including faculty — will increase their use of data. 

Second, the increased use of data to inform teaching and planning at the colleges 
is promising and is consistent with the goals of ATD. The survey findings related to 
faculty members’ use of data to inform teaching decisions are consistent with previous 
findings by Jenkins et al. (2012) that ATD colleges in Washington State that participated 
in the survey were more likely than Round 1 ATD colleges to implement changes in 
instruction, as opposed to student support services. Faculty members’ focus on 
instructional changes is also encouraging because improving student completion rates 
will likely require systemic changes to academic programs, which can only be 
accomplished with the active participation of faculty — ideally, those who are using data 
on their students to make decisions about which changes to implement. 

Overall, we found evidence of practices and perceptions related to data use at 
these colleges that were consistent with, and possibly the result of, participation in ATD. 
There was broad use of data by administrators and of certain types of data by faculty and 
student services staff, but there remain opportunities for increasing the use of student data 
in the Washington State ATD colleges. More work may be necessary to achieve the 
broad engagement and systemic improvement that ATD encourages. To further promote 
the use of data in support of improved student success, ATD colleges in Washington 
State and elsewhere should consider ways to better connect the data collected and 
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reported to the primary focus of faculty on instruction, and they should consider ways to 
engage student services staff more in the use of data on student progression, given their 
interest in student retention. Nonetheless, we consider the increase and maintenance of 
practices that support data use between the two waves of the survey to reflect a growing 
culture of evidence in the ATD colleges surveyed here. 
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